Evaluation of an NLG System using Post-Edit Data: Lessons Learnt
نویسندگان
چکیده
Post-editing is commonly performed on computer-generated texts, whether from Machine Translation (MT) or NLG systems, to make the texts acceptable to end users. MT systems are often evaluated using post-edit data. In this paper we describe our experience of using post-edit data to evaluate SUMTIME-MOUSAM, an NLG system that produces marine weather forecasts.
منابع مشابه
Evaluating an NLG System using Post-Editing
Computer-generated texts, whether from Natural Language Generation (NLG) or Machine Translation (MT) systems, are often post-edited by humans before being released to users. The frequency and type of post-edits is a measure of how well the system works, and can be used for evaluation. We describe how we have used post-edit data to evaluate SUMTIME-MOUSAM, an NLG system that produces weather for...
متن کاملLessons from Deploying NLG Technology for Marine Weather Forecast Text Generation
SUMTIME-MOUSAM is a Natural Language Generation (NLG) system that produces textual weather forecasts for offshore oilrigs from Numerical Weather Prediction (NWP) data. It has been used for the past year by Weathernews (UK) Ltd for producing 150 draft forecasts per day, which are then post-edited by forecasters before being released to end-users. In this paper, we describe how the system works, ...
متن کاملLessons learnt from errors in radiotherapy centers
Background: The purpose of this work is to discover and analyze errors and incidents in some radiotherapy centers, and to introduce methods that could reduce their occurrences, especially those which had happened due to the use of improper and inadequate equipment. This work is a first step toward clarifying the role of education in a risk-conscious culture, and changing the attitude of radioth...
متن کاملThe Importance of Narrative and Other Lessons from an Evaluation of an NLG System that Summarises Clinical Data
The BABYTALK BT-45 system generates textual summaries of clinical data about babies in a neonatal intensive care unit. A recent taskbased evaluation of the system suggested that these summaries are useful, but not as effective as they could be. In this paper we present a qualitative analysis of problems that the evaluation highlighted in BT-45 texts. Many of these problems are due to the fact t...
متن کاملWhat is in a text and what does it do: Qualitative Evaluations of an NLG system - the BT-Nurse - using content analysis and discourse analysis
Evaluations of NLG systems generally are quantiative, that is, based on corpus comparison statistics and/or results of experiments with people. Outcomes of such evaluations are important in demonstrating whether or not an NLG system is successful, but leave gaps in understanding why this is the case. Alternatively, qualitative evaluations carried out by experts provide knowledge on where a syst...
متن کامل